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(54) Color image pickup device 

(57) In a method tor operating a data processing 
system to generate a second image from a first image 
having partially sampled colour values at each pixel, the 
first image including a two-dimensional array of pixel 
values, each of the pixel values corresponding to the 
light intensity in one of a plurality of spectral bands at a 
location in the first image, and the second image in- 
cludes a second two-dimensional array of colour vec- 
tors, each colour vector having a light intensity value for 
each of the spectral bands, and there being one such 
vector corresponding to each location having a pixel val- 
ue in the first image, one component of the vector being 
equal to the pixel value in the first image at that location, 
the missing colour components are computed at each 
location. The method begins by providing an estimate 
for each component that is not equal to one of the pixel 
values from the first image for each vector. An updated 
estimate for each of the estimates is then generated by 
optimising a weighted sum of first and second functions. 
The first function measures the degree of roughness of 
the second image, and the second function measures 
the degree to which the vectors change direction be- 
tween neighbouring locations in the second image. 
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Description 

[0001] The present invention concerns a method for handling images when operating a data processing system. It 
relates to digital cameras, and more particularly to an improved method for converting data from a camera sensor to 
5 a colour image. 

[0002] A digital color image usually consists of an array of pixel values representing the intensity of the image at 
each point on a regular grid. Typically, three colors are used to generate the image. At each point on the grid the 
intensity of each of these colors is specified, thereby specifying both the intensity and color of the image at that grid point. 
[0003] Conventional color photography records the relevant image data by utilizing three overlapping color sensing 
10 layers having sensitivities in different regions of the spectrum (usually red, green, and blue). Digital cameras, in contrast, 
typically utilize one array of sensors in a single 'layer 0 . 

[0004] When only one sensor array is used to detect color images, only one color may be detected at any given 
sensor location. As a result, these sensors do not produce a color image in the traditional sense, but rather a collection 
of individual color samples, which depend upon the assignment of color filters to individual sensors. This assignment 
is is referred to as the color filter array (CFA) or the color mosaic pattern. To produce a true color image, with a full set 
of color samples (usually red, green and blue) at each sampling location, a substantial amount of computation is re- 
quired to estimate the missing information, since only a single color was originally sensed at each location in the array. 
This operation is typically referred to as "demosaicing". 

[0005] To generate the missing information, information from neighboring pixels in the image sensor must be used. 

20 A number of algorithms have been put forward in an attempt to provide the missing information while minimizing artifacts 
resulting from the estimation process. The simplest algorithms interpolate the sensor data from like color sensors to 
provide the missing information. These algorithms treat the red sensors as being independent from the green sensors, 
and so on. To provide a red value at a given location, the values measured by the red sensors in the region of that 
location are interpolated. This approach requires that the image be low-pass filtered. Such filtering reduces the image 

2S resolution below the pixel resolution of the underlying sensor array. This lost resolution cannot be recovered. 

[0006] In addition, systems that treat the color sensors independently can generate color artifacts in some images. 
These artifacts express themselves as streaks of false colors in the restored image, and are especially apparent around 
boundaries between different objects in the image and in textured areas. 

[0007] To avoid the loss in resolution discussed above, less aggressive optical low-pass filtering is used in some 
30 higher-end cameras. However, in such systems, the color sensors may no longer be treated as independent. For 
example, Wober, et al. (U.S. Patent 5,475,769) describe a method for generating the missing color information by 
computing a weighted average of the pixel values in the neighborhood of the pixel whose missing color information is 
being computed. This method weights values from all of the color sensors, not just the color being reconstructed. 
However, even this approach leaves much to be desired since it utilizes one set of weights for all images, and hence, 
35 there are always images in which unwanted artifacts are generated. 

[0008] Broadly, it is the object of the present invention to provide an improved image processing methodfor converting 
data from a pixel array having non-overlapping sensors to a fully sampled digital image. 

[0009] It is a further object of the present invention to provide a conversion method that does not generate the color 
artifacts discussed above. 

40 [0010] These and other objects of the present invention will become apparent to those skilled in the art from the 
following detailed description of the invention and the accompanying drawings. 

[0011] The present invention is a method for operating a data processing system to generate a second image from 
a first image having partially sampled color values at each pixel. The first image includes a two-dimensional array of 
pixel values, each of the pixel values corresponding to the light intensity in one of a plurality of spectral bands at a 

46 location in the first image. The second image includes a second two-dimensional array of color vectors. Each color 
vector has a light intensity value for each of the spectral bands. There is one such vector corresponding to each location 
having a pixel value in the first image. One component of the vector is equal to the pixel value in the first image at that 
location. The present invention computes the missing color components at each location. The method begins by pro- 
viding an estimate for each component that is not equal to one of the pixel values from the first image for eac v:ctor. 

so An updated estimate for each of the estimates is then generated by optimizing a weighted sum of first ano r ^cond 
functions. The first function measures the degree of roughness of the second image, and the second function measures 
the degree to which the vectors change direction between neighboring locations in the second image. 
[001 2] Figure 1 illustrates the demosaicing problem as applied to a typical image taken with an image sensor having 
a repeating 2x2 pattern. 

55 [0013] The method of the present invention may be applied to any color-sampling device; however, to simplify the 
following discussion, the method of the present invention will be explained in terms of an image sensor having a sensor 
pattern that is constructed by repeating a kernel of sensing elements. For example, one common image sensor array 
is based on the Bayer pattern which is generated by repeating a 2x2 sensor array kernel having two green sensors, 
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one red sensor, and one blue sensor. This pattern is shown in Figure 1 at 10. The kernel is shown at 12. 
[0014] The present invention converts the data from such an array into an augmented array shown at 10* in which 
each pixel has intensity values representing each of three colors. The missing colors are shown in a smaller typeface 
in the figure. The individual sensor locations are assigned an address of the form (am), where n is the row number 

s and m is the column number in which the sensor is located. 

[001 5] There are an infinite number of patterns 1 0' that satisfy the constraint that the measured values are as shown 
in 10. Hence, some additional constraints must be applied in generating the sensor values. The present invention 
utilizes two constraints in determining the missing color values. The first constraint is based on the observation that 
most images of interest to human observers are locally smooth. Hence, this constraint penalizes choices for the un- 

10 known color values that lead to abrupt changes in intensity in the image. The constraint utilizes a measure of the 
roughness of the image. For example, the function 

JJ</£ + fl*+G^+G? + S^ + Efydxdy (1) 

75 

where, R* denotes the derivative of R with respect to x, etc., is a measure of the roughness of the image over the 
region of integration. Here R, B, G are the red, green, and blue color channels of the image. In the preferred embodiment 
of the present invention, the discrete analog of Eq. (1 ) is used to measure the roughness of the image. The roughness 
is measured over a 2 pixel range about the point being computing. In the preferred embodiment of the present invention, 
20 a function, denoted by Rough is defined as follows: 

Rough(n,m)= (r(n+ 1,m) - r(n-1,m)) 2 + (r(n,m+ 1) - r(n,m-1)f + (g(n+ 1,m) -g(n - 
2S l,m)f + (g(n,m+ 1) '9(n,m-1)f + b(n+ 1,m) - b(n-1,m)f + (b(n,m+ 1) - b(n,m~1)f 

[0016] The second constraint is based on the observation that most images of interest are locally smooth in color 
as well. Hence, this constraint penalizes choices for unknown color values that lead to abrupt changes in color. This 
constraint is implemented by defining a function that measures the degree to which adjacent pixels have color vectors 
30 (R, G, B) that have similar directions in color space. This function will be referred to as the Color Compatibility Function 
(CCF). 

[0017] Formally, the CCF is defined as follows. For each pixel (viewed as a color vector), compute its vector product 
with all its neighbors; this gives 8 vectors. The CCF function is the sum of the squared norms of all these vectors. The 
CCF function at pixel (n,m) is given by: 

35 

CCF(n, m) = ||f(n, m), g(n,m), fc(n,m))X(r(n-1,m-1), g(n-1,m-1),b(n-1, m-1))|| 2 + 
||(/t n,m) t g( n,m), b( n,m))X( r(n,/n-1 ), g( n,m-1 ). fc(n,m-1 ))[| 2 + 

40 2 

\\(r(n,m), 9 (n,4b(n,m))X(r(n+1 ) m-1) l 9(n+1 l m-1),d(n+1, m-1))H + 
\\(r(n,m) t g(n,ml b( n,m))X(f(n-1 , m),p( n-1 , m), fc(n-1 , m))|| 2 + 

\\{r{n t m) t g(n,m), b(n,m))X(r{n-'\ , m+1), p(n-1,/D+1), *>(n-1 ,/n+1))fl 2 + 
||(i<n J m) l p(n,m)6(n I m))X(r(n f m4.1).fl(n f m+1) I 6(am+1))|f + 

50 2 

\\{r{n f m),g(am). b(n t m))^(r(n^ t m+1), p(n+1, m+1), b(n+1, m+1))|| 
Here. X denotes the vector product. 

[0018] The missing color values are defined by finding the values that minimize a weighted sum of the Rough and 
ss CCF functions over the image. That is 
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Q = £Rough(n,m)+ X£cCF(n,m) (2) 



where, X is a positive constant. In the preferred embodiment of the present invention, the missing color values are 
determined by utilizing the conventional Gauss-Seidel iterative method. It should be noted that only the color values 
that were not measured by the digital camera are updated. In practice, the process converges after 4 or 5 iterations. 
10 it has also been found experimentally that the final result is not a sensitive function of the value of X over some minimum 
value. In practice a lvalue of 1.0 provides satisfactory results. 

[001 9] The above examples have utilized particular functions for measuring the roughness of the image and the rate 
of change of the color vectors in each region. However, it will be obvious to those skilled in the art from the preceding 
discussion that other functions may be utilized. 
is [0020] The method of the present invention is preferably practiced on a general purpose digital computer. However, 
other computing platforms with or without specialized hardware may be utilized. 

[0021] The above-described embodiments of the present invention utilize a procedure that searches for a minimum 
of a function consisting of a weighted sum of two functions, one measuring the roughness of the image and one meas- 
uring the rate of change of the color vectors across the image. However, other functional representations and a pro- 
20 cedure that operates by maximizing analogous functions can also be utilized. 

[0022] Various modifications to the present invention will become apparent to those skilled in the art from the fore- 
going description and accompanying drawings. Accordingly, the present invention is to be limited solely by the scope 
of the following claims. 
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Claims 



1 . A method for operating a data processing system to generate a second image from a first image, said first image 
comprising a two-dimensional array of pixel values, each of said pixel values corresponding to the light intensity 

30 in one of a plurality of spectral bands at a location in said first image, and said second image comprising, a second 

two-dimensional array of color vectors, each of said vectors having a light intensity value for each of said spectral 
bands, there being one such vector corresponding to each location having a pixel value in said first image J one 
component of said vector being equal to said pixel value in said first image at that location, said method comprising 
the steps of: 

35 

providing an estimate for each component that is not equal to one of said pixel values from said first image 
for each vector; and 

generating an updated estimate for each of said estimates by optimizing a weighted sum of a first function 
40 that measures the degree of roughness of said second image and a second function that measures the degree 

to which said vectors change direction between neighboring locations in said second image. 

2. The method of Claim 1 wherein said step of generating an updated estimate comprises optimizing said weighted 
sum using a Gauss-Seidel iterative method. 



3. The method of Claim 1 wherein said second function comprises a sum of the magnitude of the vector product of 
each of said vectors adjacent to that vector in said second two-dimensional array. 

4. A method according to any preceding claim, wherein the method is employed in a digital camera. 

5. A digital camera employing the method of any preceding claim. 
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(57) In a method for operating a data processing 
system to generate a second image from a first image 
having partially sampled colour values at each pixel, the 
first image including a two-dimensional array of pixel 
values, each of the pixel values corresponding to the 
light intensity in one of a plurality of spectral bands at a 
location in the first image, and the second image in* 
eludes a second two-dimensional array of colour vec- 
tors, each colour vector having a light intensity value for 
each of the spectral bands, and there being one such 
vector corresponding to each location having a pixel val- 
ue in the first image, one component of the vector being 
equal to the pixel value in the first image at that location, 
the missing colour components are computed at each 
location. The method begins by providing an estimate 
for each component that is not equal to one of the pixel . 
values from the first image for each vector. An updated 
estimate for each of the estimates is then generated by 
optimising a weighted sum of first and second functions. 
The first function measures the degree of roughness of 
the second image, and the second function measures 
the degree to which the vectors change direction be- 
tween neighbouring locations in the second image. 
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